Skip to content

HDFS-16267. Make hdfs_df tool cross platform#3542

Merged
goiri merged 3 commits intoapache:trunkfrom
GauthamBanasandra:df-x-platform
Oct 13, 2021
Merged

HDFS-16267. Make hdfs_df tool cross platform#3542
goiri merged 3 commits intoapache:trunkfrom
GauthamBanasandra:df-x-platform

Conversation

@GauthamBanasandra
Copy link
Member

@GauthamBanasandra GauthamBanasandra commented Oct 11, 2021

Description of PR

The source files for hdfs_df uses getopt for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with boost::program_options to make this cross platform.

How was this patch tested?

Ran hdfs_df locally and verified that it's giving the right output -
image

Also, the unit tests ran fine -
image

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@hadoop-yetus
Copy link

hadoop-yetus commented Oct 11, 2021

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 44m 22s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
-1 ❌ mvninstall 0m 26s /branch-mvninstall-root.txt root in trunk failed.
-1 ❌ compile 0m 23s /branch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt hadoop-hdfs-native-client in trunk failed.
-1 ❌ mvnsite 0m 24s /branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-native-client.txt hadoop-hdfs-native-client in trunk failed.
+1 💚 shadedclient 2m 5s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 23s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-native-client.txt hadoop-hdfs-native-client in the patch failed.
-1 ❌ compile 0m 23s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt hadoop-hdfs-native-client in the patch failed.
-1 ❌ cc 0m 23s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt hadoop-hdfs-native-client in the patch failed.
-1 ❌ golang 0m 23s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt hadoop-hdfs-native-client in the patch failed.
-1 ❌ javac 0m 23s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt hadoop-hdfs-native-client in the patch failed.
+1 💚 blanks 0m 0s The patch has no blanks issues.
-1 ❌ mvnsite 0m 22s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-native-client.txt hadoop-hdfs-native-client in the patch failed.
+1 💚 shadedclient 1m 12s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 0m 23s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt hadoop-hdfs-native-client in the patch failed.
+0 🆗 asflicense 0m 23s ASF License check generated no output?
51m 34s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/1/artifact/out/Dockerfile
GITHUB PR #3542
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux 332c4f22354c 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 463c7b93cabc9b10286d8dd2bd66b9d378b3bd8c
Default Java Red Hat, Inc.-1.8.0_302-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/1/testReport/
Max. process+thread count 46 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/1/console
versions git=2.9.5 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Oct 12, 2021

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 0s Docker mode activated.
-1 ❌ patch 0m 19s #3542 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.
Subsystem Report/Notes
GITHUB PR #3542
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/2/console
versions git=2.17.1
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@GauthamBanasandra
HDFS-16267. Make hdfs_df tool cross platform
bf5ae23 
* The source files for hdfs_df uses
   getopt for parsing the command
   line arguments.
* getopt is available only on Linux
   and thus, isn't cross platform.
* We need to replace getopt with
   boost::program_options to make
   this cross platform.
@GauthamBanasandra
Trigger CI
6f4caee
@GauthamBanasandra
Revert "Trigger CI"
1569657 
This reverts commit 2de1aa063d9deeae156929b80a5b4988e1ac41a8.
@hadoop-yetus
Copy link

hadoop-yetus commented Oct 12, 2021

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 2s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 1s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 33m 17s trunk passed
+1 💚 compile 2m 41s trunk passed
+1 💚 mvnsite 0m 30s trunk passed
+1 💚 shadedclient 55m 3s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 17s the patch passed
+1 💚 compile 2m 44s the patch passed
+1 💚 cc 2m 44s the patch passed
+1 💚 golang 2m 44s the patch passed
+1 💚 javac 2m 44s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 21s the patch passed
+1 💚 shadedclient 19m 13s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 34m 2s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 34s The patch does not generate ASF License warnings.
115m 35s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/artifact/out/Dockerfile
GITHUB PR #3542
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux 8d72fbdcca7b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1569657
Default Java Red Hat, Inc.-1.8.0_302-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/testReport/
Max. process+thread count 543 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/console
versions git=2.9.5 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Oct 12, 2021

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 25m 59s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 21m 35s trunk passed
+1 💚 compile 3m 1s trunk passed
+1 💚 mvnsite 0m 45s trunk passed
+1 💚 shadedclient 43m 45s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 24s the patch passed
+1 💚 compile 2m 41s the patch passed
+1 💚 cc 2m 41s the patch passed
+1 💚 golang 2m 41s the patch passed
+1 💚 javac 2m 41s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 26s the patch passed
+1 💚 shadedclient 18m 23s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 32m 2s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 48s The patch does not generate ASF License warnings.
126m 53s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/artifact/out/Dockerfile
GITHUB PR #3542
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux 23df25a51f88 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1569657
Default Java Red Hat, Inc.-1.8.0_302-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/testReport/
Max. process+thread count 737 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/console
versions git=2.27.0 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Oct 12, 2021

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 11m 9s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 24m 57s trunk passed
+1 💚 compile 2m 38s trunk passed
+1 💚 mvnsite 0m 27s trunk passed
+1 💚 shadedclient 52m 57s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 17s the patch passed
+1 💚 compile 2m 28s the patch passed
+1 💚 cc 2m 28s the patch passed
+1 💚 golang 2m 28s the patch passed
+1 💚 javac 2m 28s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 19s the patch passed
+1 💚 shadedclient 24m 47s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 31m 27s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 32s The patch does not generate ASF License warnings.
126m 14s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/artifact/out/Dockerfile
GITHUB PR #3542
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux 2190aed5f900 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1569657
Default Java Debian-11.0.12+7-post-Debian-2deb10u1
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/testReport/
Max. process+thread count 749 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/console
versions git=2.20.1 maven=3.6.0
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Oct 12, 2021

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 46s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 20m 34s trunk passed
+1 💚 compile 2m 56s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 2m 52s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 mvnsite 0m 27s trunk passed
+1 💚 shadedclient 45m 14s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 16s the patch passed
+1 💚 compile 2m 42s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 cc 2m 42s the patch passed
+1 💚 golang 2m 42s the patch passed
+1 💚 javac 2m 42s the patch passed
+1 💚 compile 2m 46s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 cc 2m 46s the patch passed
+1 💚 golang 2m 46s the patch passed
+1 💚 javac 2m 46s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 20s the patch passed
+1 💚 shadedclient 18m 17s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 31m 53s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 34s The patch does not generate ASF License warnings.
105m 5s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/artifact/out/Dockerfile
GITHUB PR #3542
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux 64636831de79 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1569657
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/testReport/
Max. process+thread count 750 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@goiri goiri merged commit 107fe22 into apache:trunk Oct 13, 2021
@GauthamBanasandra GauthamBanasandra deleted the df-x-platform branch October 13, 2021 16:46
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
@GauthamBanasandra
HDFS-16267. Make hdfs_df tool cross platform (apache#3542)
186b4ff
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@goiri goiri goiri approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@GauthamBanasandra @hadoop-yetus @goiri